Cutting and pasting of moving objects to and from video sequences has many applications in the field of video processing. Digital segmentation of objects, which allows such cutting and pasting, has become an increasingly popular research area in video processing.
Conventionally, cutting and pasting of video objects has been performed by chroma keying, which is also referred to as blue screen matting or “blue-screening.” In chroma keying, foreground objects are video recorded in front of a solid-colored background, usually blue or green, and are then separated from the background using matting techniques that take advantage of the known background color. The simplicity of these techniques enables rapid foreground separation. Some systems even compute chroma keying in real time. However, these methods are limited to simple backgrounds of a solid color. Errors often occur when foreground objects contain colors similar to the background.
Some conventional approaches for video object cutout involve silhouette tracking. Although these existing methods can be applied to general backgrounds, to yield greater robustness in the tracking process, they use smooth curves that imprecisely and coarsely represent object boundaries. Since a coarse boundary descriptor cannot capture the fine details of a silhouette, these techniques are inadequate for most cut-and-paste applications. These rough boundary techniques can be interactively refined by auto keying, which provides a user interface for detailed boundary adjustment through spline editing. However, since each video frame must be individually modified by the user, the manual work needed to properly delineate the boundary details is prohibitive.
Recently, conventional video matting techniques have relaxed the solid color background requirement to allow smooth color changes instead of a single solid background color. The success of video matting in such scenarios depends on various techniques, such as how accurately trimaps can be propagated and how well Bayesian matting performs in each individual frame.
Video matting when the background is complex has two main difficulties for general video sequences. First, many videos contain fast motion, morphing silhouettes, and often-changing topologies, which are very challenging for state-of-the-art optical flow algorithms to bidirectionally propagate trimaps. Second, even if accurate trimaps can be obtained with considerable user interaction, the Bayesian matting technique often produces unsatisfactory results when the foreground/background contains complex textures or the foreground colors are similar to the background colors. In other words, when the background is more complex than just a simple solid color, then automatically determining where the visual edges of a video object are as the video object changes and moves during video play is a sophisticated and processor-intensive task.